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POREVORD 


The  U.S.  Army  Research  Institute,  Fort  Leavenworth  Field  Unit,  conducts  a 
systems  and  training  research  program  that  supports  the  Combined  Arms  Center 
(CAC) . 

For  several  years  the  Fort  Leavenworth  Field  Unit  has  been  involved  in 
research  to  support  the  development  and  use  of  computer-driven  battle  simula¬ 
tions  for  training  battalion  through  division  command  groups.  A  pressing 
problem  with  the  current  generation  of  battle  simulations  is  the  burden  placed 
on  controllers,  who  must,  in  addition  to  many  other  duties,  generate  a  stream 
of  realistic  tactical  messages.  This  research  will  contribute  to  identifica¬ 
tion  of  requirements  for  an  automated  system  to  translate  simulation  output 
into  more  realistic  messages,  thereby  improving  training  realism  and  reducing 
controller  overload. 

This  research  was  an  exploratory  effort  funded  under  Research  Task  1301. 
It  compares  the  ability  of  experienced  officers  to  identify  messages  tran¬ 
scribed  from  command  post  exercises  with  their  ability  to  identify  similar 
messages  transcribed  from  field  exercises.  It  also  investigates  which  char¬ 
acteristics  of  messages  are  most  salient  in  identification. 


Technical  Director 
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IDENTIFICATION  OP  COMMAND  POST  EXERCISES  (CPX)  AND  FIELD  TRAINING  EXERCISES 
(FTX)  MESSAGES 


EXECUTIVE  SUMMARY 


Requirement: 

Command  groups  are  Increasingly  turning  to  computer-driven  command  post 
exercises  (CPX)  as  an  economical  and  convenient  training  mode.  The  overall 
realism  of  this  method  of  training  relies  heavily  on  the  capabilities  of  the 
control  staff,  vho  must  play  the  roles  of  key  personnel  in  the  units  at  eche¬ 
lons  above,  belov,  and  adjacent  to  the  training  audience.  The  controllers 
must  generate  an  information  stream  of  realistic  tactical  messages  vhile 
attending  to  a  myriad  of  other  duties  that  place  demands  on  their  time.  Fur¬ 
thermore,  this  task  is  not  veil  supported  by  the  current  generation  of  battle 
simulations.  Automating  the  production  of  realistic  messages  is  one  solution 
to  this  problem.  This  study  vas  designed  to  provide  information  that  would 
contribute  to  identification  of  the  requirements  for  a  system  to  automate  the 
translation  of  simulation  output  into  realistic  messages. 


Procedure: 

Thirty-one  company  grade  combat  arms  officers  participated  in  this  study. 
Participants  were  presented  vith  48  stimulus  messages  in  vritten  text  (24  CPX 
and  24  FTX)  and  asked  to  identify  the  training  environment  in  which  they 
originated.  Each  message  vas  further  evaluated  on  four  semantic  differential 
scales:  vordy-succlnct,  vague-precise,  worthless -valuable,  and  excited-calm. 
The  participants  vere  also  asked  to  respond  to  a  questionnaire  that  assessed 
their  beliefs  concerning  characteristics  of  CPX  and  FTX  messages  and  that 
obtained  participant  demographic  information. 


Findings: 

The  majority  of  the  officers  vho  participated  in  this  study  expressed  the 
belief  that  CPX  and  FTX  messages  could  be  distinguished  based  upon  certain 
characteristics.  However,  their  actual  performance  in  classifying  messages 
vas  better  than  chance  but  less  accurate  than  anticipated. 

The  participants  vere  also  questioned  concerning  FTX  and  CPX  message 
characteristics.  The  message  characteristics  believed  to  be  most  salient  vere 
as  follows:  CPX  messages  are  longer  than  FTX;  also  CPX  messages  are  less  emo¬ 
tional  and  contain  more  accurate  information  than  FTX  messages.  On  the  seman¬ 
tic  differential  scales,  participants  rated  CPX  messages  as  more  succinct, 
precise,  and  valuable  than  FTX  messages.  The  major  difference  between  the  two 
data  sources  vas  in  message  length.  Participants  expressed  the  belief  that 
CPX  messages  are  longer  than  FTX  messages  but  rated  CPX  messages  as  more  suc¬ 
cinct  on  the  semantic  differential  scale. 


vii 


Utilization  of  Findings: 

The  information  obtained  in  this  study  will  contribute  to  identification 
of  requirements  for  an  automated  system  to  translate  simulation  output  into 
more  realistic  messages. 

The  characteristics  that  appear  to  be  salient  in  message  identification 
should  be  used  to  translate  CPX  into  FTX-like  messages.  These  translated 
messages  should  then  be  tested  to  determine  if  experienced  officers  perceive 
them  as  realistic  FTX  messages. 
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IDENTIFICATION  OF  COMMAND  POST  EXERCISES  (CPX)  AND  FIELD  TRAINING 

EXERCISES  (FTX)  MESSAGES 

Introduction 

2 

AirLand  Battle  doctrine  relies  heavily  on  command  and  control  (C  )  to  ensure 
battlefield  success.  Commanders  and  their  staffs  oust  perform  their  u  func¬ 
tions  in  a  synchronized,  coordinated  manner  in  a  highly  dynamic  environment. 
Their  successful  battlefield  performance  depends  upon  how  well  they  have  been 
trained;  however,  the  opportunity  for  training  i6  relatively  infrequent. 

Field  exercises  (FTXs)  provide  good  ''hands-on"  training  opportunities  for 
command  teams.  However,  training  in  this  mode  is  very  expensive.  Furthermore, 
FTX  training  is  extremely  complicated  to  organize  and  execute,  which  further 
restricts  the  frequency  with  which  FTX  training  is  conducted. 

Command  groups  are  increasingly  turning  to  computer— driven  command  post 
exercises  (CPXs)  as  an  economical  and  convenient  training  mode.  These  training 
systems,  such  as  ARTBASS1,  enable  commanders  and  their  staffs  to  practice  their 
command  and  control  functions  under  simulated  battle  conditions.  However,  the 
overall  effectiveness  and  realism  of  this  method  of  training  relies  heavily 
upon  the  capabilities  of  the  control  staff. 

Controllers  must  play  the  roles  of  key  personnel  in  the  unlt6  at  echelons 
above,  below,  and  adjacent  to  the  training  audience.  This  role  playing  func¬ 
tion  provides  the  training  audience  with  insulation  from  the  computer  and  also 
provides  them  with  communication  training  (Solick  and  Lussler,  1988). 

Controllers  must  also,  among  other  duties,  generate  am  information  stream  of 
realistic  tactical  messages  based  on  the  simulation  output.  This  task  is  not 
well  supported  by  the  current  generation  of  battle  simulations.  For  instance, 
simulation  output  compromises  message  traffic  realism  by  providing  too  much  in¬ 
telligence  information  and  too  little  detail.  An  illustrative  example  of  this 
is  provided  by  Solick,  et  al  (1988): 

"For  example,  a  sensor  report  might  include  a  complete 
opposing  force  unit  designation  or  an  exact  center-of-mass 
location,  whereas  a  realistic  report  might  provide  an  estimate 
of  the  number  of  vehicles  sighted,  type  of  vehicles,  approximate 
location  and  direction."  (p.  24) 

The  translation  of  the  computer  output  to  tactical  messages  is  performed  by 
controllers  in  addition  to  the  other  numerous  tasks  which  place  demands  upon 
their  time,  such  as  directing  simulated  subordinates,  keeping  themselves  ap¬ 
prised  of  the  tactical  situation,  and  coordinating  with  other  control  elements. 
Automating  the  production  of  realistic  message  traffic  is  one  solution  to  this 
problem. 

This  report  examines  whether  Army  officers  who  have  experienced  both  FTX  and 
CPX  training  at  the  battalion  level  can  distinguish  which  training  environment 
produced  a  set  of  stimulus  messages  and  investigates  the  characteristics  of  the 
messages  that  contribute  to  successful  identification.  This  Information  will 
contribute  to  the  identification  of  requirements  for  an  automated  system  to 
translate  computer  output  into  realistic  messages. 


lARTBASS  -  Army  Training  Battle  Simulation  System 
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Method 


Participants:  The  participants  in  this  study  were  31  company  grade  combat  arms 
officers  (6  2Lt.;  16  lLt.J  9  Cpt.)  who  volunteered  their  time  during  May,  1988. 

Data  Collection  Software  and  Procedures:  Data  for  this  study  were  obtained 
through  the  use  of  MSGJUDGE  software  developed  by  Vruels  Research  Corporation 
under  contract  to  the  Army  Research  Institute  (see  Sollck,  Llbehaber,  Ober- 
mayer,  Linvllle,  and  Obermayer,  1989).  MSGJUDGE  is  a  measurement  Instrument 
for  the  computer  administration  of  a  set  of  written  messages  from  two  or  more 
different  environments  to  a  panel  of  expert  judges.  Written  messages  were  used 
In  this  study  rather  than  actual  taped  oral  messages  because  of  the  difficulty 
in  obtaining  taped  messages  which  were  uniform  in  quality  from  both  training 
environments.  MSGJUDGE  collects  data  in  the  form  of  responses  to  semantic 
scales,  forced-choice  questions,  and  a  summary  questionnaire.  The  software  was 
pilot  tested  prior  to  use  in  this  study.  Information  concerning  the  pilot  data 
collection  is  contained  in  Solick,  et  al,  (1989). 

In  the  present  study,  46  messages  were  presented  to  each  participant.  The 
original  pool  of  292  messages  was  composed  of  73  messages  in  each  of  the  four 
categories  (CPX  enemy  operations,  CPX  friendly  operations,  FTX  enemy  opera¬ 
tions,  and  FTX  friendly  operations).  Twelve  messages  were  randomly  selected 
from  the  73  in  each  of  these  groups  for  presentation  to  subjects.  A  list  of 
the  48  items  which  were  administered,  and  the  classification  of  each  item,  is 
attached  as  Appendix  A. 

For  administration,  a  participant  was  presented  with  a  stimulus  message 
which  was  displayed  across  the  top  of  his  computer  screen;  each  of  four  bi¬ 
polar  scales  successively  occupied  the  screen  below.  The  four  semantic  scales 
used  for  this  study  were:  wordy-succinct,  vague-precise,  worthless-valuable, 
and  excited-calm.  Rating  was  on  a  7  point  scale  ranging  from  -3  to  +3.  For 
any  given  message,  all  of  the  semantic  scales  were  administered  one  at  a  time 
until  all  measurement  was  complete  for  that  message.  The  following  example 
shows  the  configuration  of  the  screen  for  the  semantic  measurement. 


Message:  We  have  detected  2  BMP  at  256  947. 

-3  -2  -1  0  +1  +2  +3 

Wordy  ::::::::  Succinct 

f 

Pointer 


The  next  screen  showed  the  same  message  with  another  semantic  scale,  and 
thl6  was  repeated  until  all  four  scales  were  administered.  At  that  time,  a  new 
message  was  presented  and  the  scales  were  repeated. 

When  all  messages  had  been  presented  for  semantic  measurement,  the  same 
messages  were  again  presented  in  a  forced-choice  format  (CPX  or  FTX).  The 
screen  configuration  for  this  segment  of  the  measurement  is  shown  in  the  fol¬ 
lowing  example: 
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Message:  We  have  detected  2  BMP  at  256  94  7 


In  this  message  from  a  CPX  or  FTX? 
Enter  first  letter  from  choice: 


Finally,  the  participants  were  presented  with  questions  which  obtained  demo** 
graphic  information  and  assessed  beliefs  concerning  various  characteristics  of 
FTX  and  CPX  messages. 


Results 

Message  Identification: 

Participants  were  asked  to  identify  whether  each  of  48  messages  originated 
in  an  FTX  or  a  CPX  environment.  The  number  of  the  messages  identified  cor¬ 
rectly  (out  of  a  total  of  1488)  for  both  categories  and  subcategories  is  pre¬ 
sented  in  Table  1. 


Table  1 


Message  Identification  -  Number  Correct 


Friendly 

Total 

FTX 

235 

254 

489 

CPX 

165 

210 

375 

Total 

400 

464 

864 

A  two-way  analysis  of  variance  of  the  number  of  correct  identifications 
indicated  that  FTX  messages  were  correctly  identified  significantly  (p.  <.01 ) 
more  often  than  CPX  messages,  and  messages  with  friendly  operations  content 
were  correctly  identified  significantly  (p.  <.01)  more  often  than  messages  with 
eneny  information  content,  regardless  of  their  FTX  or  CPX  classification. 

Semantic  Differential  Ratings: 

Participants  were  asked  to  rate  each  message  on  four  semantic  differential 
scales:  wordy-succinct,  vague-precise,  worthless-valuable,  and  excited-calm. 
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These  ratings  were  obtained  in  order  to  ascertain  if  these  semantic  character¬ 
istics  were  useful  in  identifying  their  FTX  or  CPX  origin  and  could  be  used  to 

develop  rules  for  making  CPX  messages  more  like  FTX  messages.  The  participants 
classified  CPX  messages  as  more  succinct,  precise,  valuable  and  excited  than 
FTX  messages.  The  mean  rating  for  each  scale  is  presented  in  Table  2.  The  48 

messages  with  their  mean  rating  on  each  semantic  differential  scale  are  pre¬ 

sented  in  Appendix  A. 

A  discriminant  analysis  of  the  semantic  differential  ratings  indicated  tha*: 
three  of  the  scales  were  useful  in  predicting  classification.  These  scales,  in 
order  of  importance  of  their  contribution,  are:  vague-precise,  wordy-succinct, 
and  worthless-valuable.  The  excited-calm  scale  added  no  independently  useful 
information  to  the  equation. 


Table  2 

Semantic  Differential  Scales-Mean  Ratings 


Scale 

FTX 

CPX 

Wordy  -  Succinct 

.45 

1.58 

Vague  -  Precise 

-.33 

1.00 

Worthless  -  Valuable 

.60 

1.60 

Excited  -  Calm 

-.02 

.24 

Demographic  and  Attitude  Questions:  Participants  were  asked  to  respond  to  24 
questions  concerning  their  experience  with  FTX  and  CPX  training  environments 
and  concerning  their  attitudes  and  beliefs  pertaining  to  messages  obtained  in 
the  two  environments.  These  questions,  together  with  responses,  are  presented 
below. 

1.  Have  you  ever  participated  in  a  Command  Post  Exercise  (CPX)? 

Response:  Yes  ■  90.32 
No  -  9. 72 

2.  Have  you  ever  participated  1..  a  Field  Training  Exercise  (FIX)? 

Response:  Yes  *  100.002 

3.  Do  you  think  it  is  possible,  in  general,  to  tell  CPX  messages  from  FTX 
messages? 


Response:  Yes  ■  61.3% 
No  *  38.72 
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Do  you  consider  the  following  to  be  recognizable  differences  between  CPX 

and  FTX  messages? 

4.  Length  of  Transmission: 

Response:  Yes  *  71. OX 
No  -  29.0% 

5.  Vocabulary: 

Response:  Yes  ■  83.9% 

No  -  16.1% 

6.  Emotion  of  the  Speaker: 

Response:  Yes  ■  80.6% 

No  -  19.4% 

7.  Speaker,  i.e.  First  Person  Voice  or  Third  Person: 

Response:  Yes  ■  64.5% 

No  -  35.5% 

6.  Completeness  of  Sentences: 

Response:  Yes  «  74.2% 

No  -  25.8% 

9.  Accuracy  of  Descriptions: 

Response:  Yes  *  90.3% 

No  -  9.7% 

10.  Passive  Voice  vs  Active  Voice: 

Response:  Yes  ■  77.4% 

No  -  22.6% 

11.  Present  Tense  Speaker  vs  Past  Tense  Speaker: 

Response:  Yes  *  48.4% 

No  -  51.6% 

12.  Use  of  key  "tip-off"  words: 

Response:  Yes  ■  80.6% 

No  -  19.4% 

13.  Do  you  think  the  transmissions  will  be  longer  in  an  FTX  or  CPX? 

Response:  FTX  ■  35.5% 

CPX  -  64.5% 
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14.  Will  the  speaker  be  more  emotional  in  a  CPX  or  FTX? 

Response:  FTX  ■  93.5% 

CPX  -  6.5% 


15.  Will  the  speakers  have  a  more  varied  vocabulary  in  an  FTX  or  in  a  CPX? 

Response:  FTX  •  51.6% 

CPX  ■  48.4% 

16.  Will  the  transmission  contain  more  accurate  information  if  from  a  CPX  or 
from  a  FTX? 


Response:  FTX  ■  9.7% 

CPX  «  90.3% 

17.  Db  you  feel  that  the  quality  of  training  can  be  affected  based  on  the 
realism  of  radio  transmissions  that  describe  the  battlefield  to  the  Commander 
and  staff? 


Response:  Yes  ■  93.5% 

No  -  6.5% 

18.  Have  you  ever  participated  at  an  opposing  force  exercise  at  the  National 
Training  Center? 

Response:  Yes  *  80.6% 

No  -  19.4% 

19.  Have  you  ever  participated  in  a  CPX  using  ARTBASS? 

Response:  Yes  ■  64.5% 

No  -  35.5% 

20.  Which  environment,  CPX  or  FTX,  provides  the  best  training  for  yourself? 

Response:  FTX  ■  93.5% 

CPX  -  6.5% 

21.  Which  environment,  FTX  or  CPX,  provides  the  best  training  for  the  battal¬ 
ion  staff? 

Response:  FTX  ■  58.1% 

CPX  -  41.9% 
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22.  The  self-administered  program  you  just  completed  is  the  initial  phase  of  a 
project  to  describe  differences  in  communications  between  CPX  and  FTX.  The 
goal  of  the  project  is  to  provide  more  realistic  training  environments.  Do  you 
think  this  self-administered  message  judgment  program  will  show  those  differ¬ 
ences? 


Response:  Yes  ■  48. 4% 

No  -  51.6% 

23.  1  also  wish  to  determine  if  there  is  a  training  difference  between  CPX  and 

FTX.  1  am  going  to  use  the  differences  in  message  traffic  to  describe  that 
difference.  Do  you  think  this  program  will  help  to  identify  the  training  dif¬ 
ferences? 


Response:  Yes  ■  58.1% 

No  -  41.9% 

24.  Is  this  type  of  research ,  determining  differences  between  CPX  and  FTX 
messages  to  improve  training,  beneficial  to  the  Army? 

Response:  Yes  ■  71.0% 

No  -  29.0% 


Discussion  and  Conclusions 

As  currently  practiced,  CPX  training  does  not  provide  realistic  tactical 
messages  for  the  training  audience.  This  is  primarily  because  the  controllers, 
who  are  responsible  for  message  generation  have  too  little  time  and  too  great  a 
workload  to  adequately  produce  realistic  messages.  There  is  a  widely  hold 
view,  supported  by  the  majority  of  the  Army  officers  who  participated  in  this 
study,  that  the  quality  of  training  can  be  affected  by  the  realism  of  messages 
that  describe  the  battlefield  to  the  Commander  and  staff.  These  participants 
also  indicated  that  they  believe  that  they  can  distinguish  messages  generated 
in  a  CPX  environment  from  messages  generated  in  an  FTX  environment.  The  pur¬ 
pose  of  this  study  was  to  determine  if  CPX  and  FTX  messages  could,  in  fact,  be 
distinguished,  and  to  ascertain  which  characteristics  of  the  messages  were  most 
salient  in  this  identification.  Ultimately,  this  information  could  be  used  to 
either  automate  the  message  generation  process  or  provide  guidelines  for  con¬ 
trollers  which  would  contribute  to  the  production  of  more  realistic  messages  as 
well  as  reduce  controller  workload. 

The  results  of  this  study  indicate  that  the  participants  could  distinguish 
CPX  from  FTX  messages,  but  their  accuracy  was  somewhat  less  than  anticipated. 
Although  classification  accuracy  was  significantly  better  for  FTX  messages  than 
for  CPX  messages,  this  could  have  been  the  result  of  a  general  FTX  response 
bias  when  in  doubt  about  a  message's  true  classification. 

Nevertheless,  the  participants  expressed  strong  beliefs  concerning  the  sali¬ 
ent  characteristics  which  distinguish  CPX  messages  from  FTX  messages.  Partici¬ 
pants  expressed  the  belief  that  CPX  messages  are  longer,  less  emotional  and 
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contain  more  accurate  information  than  FTX  messages •  Additionally,  they  be¬ 
lieve  that  CPX  and  FTX  messages  can  be  distinguished  on  the  basis  of  vocabulary 
and  "tip-off"  words. 

The  analysis  of  the  semantic  differential  ratings  yielded  similar  results. 
Participants  rated  CPX  messages  as  more  succinct,  precise  and  valuable  than  FTX 
messages.  There  was  a  discrepancy  here  in  that  CPX  messages  were  rated  as  more 
succinct,  but  participants  expressed  the  belief  that  they  are  longer  than  FTX 
messages.  The  fact  that  the  calm-excited  scale  failed  to  provide  any  addi¬ 
tional  information  could  be  explained  by  the  fact  that  the  stimulus  messages 
were  presented  in  the  form  of  written  text  rather  than  spoken  form,  which  would 
provide  more  opportunity  for  projection  of  emotion. 

Although  further  research  may  be  needed  to  resolve  the  discrepancy  in  the 
results  concerning  the  saliency  of  message  length  in  message  classification, 
the  information  obtained  in  this  study  has  been  formulated  into  rules  for  gen¬ 
erating  more  realistic  CPX  messages.  These  rule-generated  messages  were  tested 
to  determine  if  participants  would  perceive  them  as  FTX  messages.  The  results 
of  this  study  are  presented  in  Lussier,  Solick,  Keene  and  Linvllle  (1989).  The 
following  example  demonstrates  how  this  transformation  process  may  work. 

Original  Alert  Message:  HA/B/3-41  visually  detected  at  NK.  320  321  1  T64. 

Transformation  Message:  I've  got  a  lone  T-64  off  to  the  North  of  my 

position.  Looks  like  it  may  be  disabled,  but  can't 
tell  for  sure. 

In  this  example,  the  CPX  simulation  alert  message  was  made  more  realistic  by 
adding  uncertainty  and  vagueness.  When  the  transformed  version  of  this  message 
was  presented  to  participants,  872  perceived  it  as  a  FTX  message  (Lussier,  et 
al  1989). 
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Some  messages  contain  both  friendly  and  enemy  in-  JA  -  wordy-succinct;  C  -  worthless-valuable; 

formation  and  were  coded  both  ways.  One  CPX  and  B  =  vague-precise;  D  ■  excited-calm 

one  FTX  message  was  randomly  selected  in  both  categories. 
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one  FTX  message  was  randomly  selected  in  both  categories. 
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